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Abstract 

For  many  years  military  command- and- control  systems  have  been  evaluated  by  the  use 
of  mission  threads.  Mission  threads  arise  from  operational  requirements  and  are  de¬ 
signed  so  that  a  system’s  response  to  a  series  of  selected  scenario  events  can  be  quan¬ 
tified.  They  may  be  used  to  answer  timeline  and  routing  questions  such  as:  ^^How  long 
does  it  take  to  complete  a  mission?”  ^^How  can  information  be  more  effectively  routed?” 
“Where  are  the  delays  in  the  mission  threads  occurring?” .  While  these  descriptive  data 
are  sufficient  for  the  decision  maker  to  judge  a  prototype  command-and-control  system 
on  a  pass/fail  basis ,  they  do  not  convey  information  about  the  performance  of  the  com¬ 
munications  subsystem;  in  particular,  they  do  not  allow  the  accurate  determination  of 
throughput  and  delays.  Mission  threads  do  not  allow  human  effects  to  be  satisfactorily 
decoupled  from  the  communications  effects,  thereby  inhibiting  the  characterization  of 
the  types  of  delays  a  system  may  be  experiencing.  An  in-house  solution  has  been  to 
generate  “messages”  that  are  simply  character  strings  of  a  specified  length  and  arrival 
rate.  Test  and  evaluation  software  inserts  the  message  strings  into  the  communications 
system,  and  allows  network  statistics  to  be  made  accessible  to  the  decision  maker  in 
an  accurate  and  timely  fashion.  This  paper  describes  the  nature  of  mission  threads, 
problems  encountered  with  their  use,  and  the  effective  use  of  message  strings  in  exper¬ 
imentation. 


Introduction 

For  many  years  the  evaluation  of  military  command-and-control  (C2)  systems  has  been 
dominated  by  the  use  of  mission  threads.  Mission  threads  are  a  time-ordered  series  of 
messages  required  to  perform  an  operational  task.  For  example,  if  there  is  an  operational 
need  to  be  able  to  place  artillery  rounds  on  a  target  within  2  minutes  of  the  forward  observer’s 
(FO)  call  for  fire,  then  the  C2  system  must  be  able  to  send  the  various  requests  and  commands 
about  the  battlefield  to  accomplish  this  task  within  the  specified  time  constraints. 
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Once  a  C2  system  is  created,  evaluation  is  required.  With  most  systems  evaluation 
consists  of  taking  the  system  to  the  field,  using  it  with  troops  in  simulated  maneuvers,  and 
having  observers  record  the  times  various  events  occur.  These  times  are  then  compared  to 
the  operational  requirements,  and  the  system  either  passes  or  fails.  Unfortunately,  testing 
in  this  manner  is  expensive  and  often  fails  to  pinpoint  bottlenecks  or  problem  areas.  Our 
premise  is  that  while  mission  threads  are  necessary  for  the  final  evaluation  of  C2  systems, 
they  are  not  sufficient  for  the  evaluation  of  today’s  digital  C2  systems. 

Decentralized  battlefield  C2  requires  reliable  and  timely  distribution  of  information. 
Presently,  information  distribution  is  limited  by  noisy  channels  and  protocols  that  do  not 
match  the  channel  to  traffic  demands,  forcing  commanders  to  make  decisions  with  out  of  date 
or  incomplete  information.  Experimental  use  of  message  strings  (arbitrary  data  used  to  pro¬ 
vide  communications  loading)  allows  for  a  much  broader-based  effort,  emphasizing  network 
adaptations  at  the  lower  levels  of  the  network  protocol  and  taking  into  account  channel  and 
workload  characteristics  but  not  application-specific  information.  This  approach  encour¬ 
ages  general  adaptations  that  improve  performance  without  actually  cutting  back  on  the 
information  delivered. 

It  is  our  premise  that  by  initially  evaluating  C2  communications  systems  using  message 
strings,  experiments  involving  the  entire  system  can  be  better  focused  on  military  factors 
with  a  validated  knowledge  of  the  service  the  communications  system  can  provide.  In  this 
paper  we  describe  several  experiments  in  which  mission  threads  were  used  as  the  primary 
evaluation  tool.  We  also  describe  an  experiment  in  which  message  strings  were  used  to 
provide  a  background  for  a  system  evaluation. 


C3  System  Evaluation  Using  Mission  Threads 


During  the  1980s,  the  U.S.  Army  Research  Laboratory  (ARL)(then  the  Ballistic  Research 
Laboratory)  participated  in  a  series  of  command,  control,  and  communications  (C3)  experi¬ 
ments.  These  experiments  were  designed  to  investigate  C3  issues  associated  with  such  things 
as  new  digital  devices  and  digital  communications  links. 

The  first  of  these,  the  Fire  Support  Team  (FIST)  Experiment  [1],  was  the  first  statis¬ 
tically  sound  test  of  its  kind  in  which  military  players  were  interfaced  with  automated  C2 
equipment,  and  researchers  were  provided  with  a  sound  database  of  actual  parameters  mea¬ 
sured  during  a  controlled  laboratory  experiment.  This  experiment  was  conducted  from  May 
through  June  1983  at  the  joint  Humaji  Engineering  Laboratory  (HEL)/Ballistic  Research 
Laboratory  (BRL)  Command  Post  Exercise  Research  Facility  (CPXRF).  The  experiment 
was  designed  to  study  the  effects  of  message  intensity  and  communications  degradation  on 
the  FIST  Headquarters’  (HQ)  ability  to  perform  fire  support  coordination  with  a  newly  de¬ 
veloped  FIST  Digital  Message  Device  (DMD).  From  an  analysis  of  variance  (ANOVA)  on 
the  data  collected,  it  was  concluded  that  communications  degradation  and  intensity  had  a 
statistically  significant  effect  on  the  message  trafl&c  through  the  FIST  HQ.  Based  on  the 
results  of  statistical  analyses,  including  ANOVA,  contingency  table  analysis,  and  modeling 
techniques,  many  software  enhancements  were  recommended  for  the  FIST  DMD. 
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In  April  and  May  of  1984,  the  FIST  Force  Development,  Testing,  and  Experimentation 
II  (FDT&E  II)  [2],  a  large-scale  field  experiment,  was  conducted  at  Fort  Riley,  KS.  This  was 
the  first  time  extensive  tactical,  digital.  Army  C3  data  were  collected  from  a  tactical  fire 
direction  system  (TACFIRE)  in  a  field  environment.  A  factorial  experiment  was  designed  to 
study  the  FIST  HQ  ability  to  perform  fire  support  coordination  under  different  forward  ob¬ 
servation  controls,  headquarters’  configurations,  and  workload  coinponents.  From  the  data 
collected,  the  FIST  HQ  ability  to  service  fire  missions  under  different  headquarters’  person¬ 
nel  configurations  was  not  statistically  significant.  However,  forward  observation  controls 
and  workload  components  had  a  significant  impact  on  the  median  time  required  to  service 
fire  missions.  Other  statistical  analyses,  including  cluster  analysis,  kernel  density  estimation 
procedures,  and  descriptive  statistics,  provided  additional  meaningful  information  about  the 
data  collected.  For  instance,  the  fire  direction  radio  net  was  used,  on  average,  23.3%  of 
the  time.  In  a  combat  environment,  this  would  provide  the  enemy  with  ample  opportunity 
to  detect  and  destroy  the  friendly  units  attached  to  this  net.  During  the  experiment,  all 
messages  were  preceded  by  a  2.1-second  preamble  (which  allowed  the  radios  to  reach  good 
transmission  operating  conditions).  If  the  preamble  could  be  reduced  to  0  seconds,  theo¬ 
retical  computations  revealed  that  the  average  net  usage  would  drop  from  23.3%  to  5.9%. 
Other  analyses  of  the  FIST  FDT&E  II  data  provided  similar-type  ideas  for  improving  field 
communications  and  equipment. 

In  December  1985,  the  Firepower  Control  Experiment,  the  second  C3  experiment  con¬ 
ducted  in  the  HEL/BRL  CPXRF  [3],  was  designed  to  investigate  the  digital  communications 
links  between  the  Field  Artillery  battery  Fire  Direction  Center  (FA  btry  FDC)  and  simulated 
155-mm  howitzer  units.  In  investigating  the  digital  communications  on  the  net  linking  the 
FA  btry  FDC  and  155-mm  howitzers,  average  net  utilization  was  26.7%.  Since  the  minimum 
message  preamble  was  used,  240  milliseconds,  this  suggested  careful  consideration  should  be 
given  to  the  radios  that  would  link  the  FA  btry  FDC  and  future  semi-autonomous  howitzers. 

A  measure  of  performance  (MOP)  is  a  response  that  is  used  to  quantify  the  effects  of 
the  factors  being  evaluated  during  an  experiment.  For  these  three  experiments,  typical 
MOPs  were  service  times  (e.g.,  elapsed  time  from  target  acquisition  until  a  fire  request 
message  is  transmitted  from  a  digital  device)  and  network  utilization.  Analysis  of  these,  and 
similar,  MOPs  enables  an  evaluator  to  state  whether  the  system  as  a  whole  meets  specific 
requirements,  but  does  not  enable  him/her  to  identify  general  problem  areas. 

Let  us  revisit  the  operational  need  to  place  artillery  rounds  on  a  target  within  a  2-minute 
window  of  opportunity  from  the  time  of  the  FO’s  call  for  fire  as  a  Fire  For  Effect  (FFE) 
Mission  Thread.  Then,  at  a  minimum,  we  need  to  address  the  events  shown  in  Table  1. 
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Table  1. — Mission  Thread  Events 


The  time  for  the  FO  to  enter  the  call  for 
fire  into  the  system 

15  seconds 

The  time  for  the  call  for  fire  to  get  from  the 

FO  to  the  Fire  Direction  Officer  (FDO) 

X 

The  time  for  the  FDO  to  process  the  call 
for  fire 

15  seconds 

The  time  for  the  call  for  fire  to  get  from 
the  FDO  to  the  battery 

Y 

The  time  for  the  gun  crew  to  lay,  load,  and 
fire  the  round(s) 

45  seconds 

The  time  of  flight  of  the  round(s) 

25  seconds 

The  times  allowed  for  each  of  these  events  may  be  determined  from  training  manuals,  field 
experiments,  and  simulations.  For  our  example,  let’s  assume  the  values  shown  previously. 
This  means  the  communications  portion  of  the  proposed  C3  system  must  be  able  to  transport 
the  required  messages  within  the  constraint  of 


X  +  Y  <=  20  seconds. 

Of  course,  “real  world”  mission  threads  axe  more  complex  than  this,  but  one  can  begin 
to  see  how  system  requirements  are  obtained. 

A  common  element  of  the  three  experiments  was  that  input  was  provided  by  tactical  sce¬ 
nario  databases  that  contained  mission  threads.  These  scripted  scenario  input  files,  coupled 
with  appropriate  timing  parameters,  were  utilized  to  maintain  the  flow  of  the  fire  mission 
segments  in  the  experiments.  Typically,  creating  scenarios  for  use  with  well-defined  experi¬ 
mental  designs  is  a  time-consuming  process.  For  example,  the  scenario  created  for  the  FIST 
Experiment  required  that  several  criteria  be  imposed  to  ensure  that  task  loading  on  the 
FIST  HQ  didn’t  vary  significantly  between  test  cells  of  the  same  message  intensity  (e.g.,  the 
ratio  of  FFE  to  Adjust  Fire  (AF)  missions  was  chosen  as  2:1,  the  number  of  adjustments  in 
each  AF  was  chosen  as  three,  and  one  fire  mission  in  each  2-hour  test  cell  was  designated  as 
urgent  rather  than  normal  priority).  It  was  also  realized  that  the  time  interval  between  fire 
missions  was  a  significant  factor  that  influenced  the  loading  on  the  FIST  HQ.  Since  timing 
wasn’t  specified  in  the  original  scenario  definition,  all  fire  mission  time-tags  were  changed 
manually  so  that  the  intervals  between  the  fire  missions  were  the  same  for  each  cell  of  the 
same  intensity.  This  procedure  alone  took  60  man-hours.  Similar  problems  presented  them¬ 
selves  in  the  development  of  tactical  scenarios  for  the  FST&E  II  and  the  Firepower  Control 
Experiment. 
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C3  System  Evaluation  Using  Message  Strings 


The  networks  that  are  of  particular  interest  to  the  Army  have  nodes  with  high  computing 
power  but  weak,  noisy,  shared  communications  links.  Our  current  approach  to  communi¬ 
cations  emphasizes  working  intelligently  at  each  node  to  limit  or  redirect  the  amount  of 
information  that  must  be  passed  along  the  channel.  Each  node  is  assumed  to  act  inde¬ 
pendently  to  improve  the  elfectiveness  of  the  information  exchange  between  nodes.  Such 
a  system  of  controls  requires  that  each  node  be  able  to;  monitor  the  network  traffic;  de¬ 
cide  whether  performance  is  inadequate;  and,  if  so,  mahe  an  adjustment  to  the  protocol  to 
improve  performance.  In  general,  the  objective  is  to  maximize  network  throughput  while 
minimizing  delay  in  the  delivery  of  information  to  the  end  user.  Protocol  parameters  such 
as  packet  size,  coding  technique,  and  channel  access  algorithm  could  be  adjusted  to  improve 
or  optimize  information  transfer.  Identifying  the  right  tool  can  often  render  a  seemingly 
impossible  problem  tractable:  message  strings  offer  us  just  such  a  tool. 

In  April  1991,  Program  Manager  -  Advanced  Field  Artillery  Tactical  Data  System  (PM 
AFATDS)  visited  BRL  to  discuss  a  Magnavox  study  that  addressed  AFATDS  communica¬ 
tions  over  Combat  Net  Radios  (CNRs)  via  the  Tactical  Fire  Direction  System  (TACFIRE) 
protocol  [4].  PM- AFATDS  wanted  an  independent  assessment  of  the  results  of  that  study, 
which  indicated  network  saturation  for  various  portions  of  their  proposed  scenarios. 

The  BRL’s  Firepower  Control  facility  was  used  to  build,  load,  and  monitor  four  AFATDS 
nodes  sharing  a  single  network  and  communicating  over  CNRs.  Each  node  consisted  of  a 
SUN  workstation  and  a  Magnavox  tactical  communications  modem  (TCM)  to  enable"  com¬ 
munications  via  SINgle- Channel  Ground  and  Airborne  Radio  System  (SINCGARS)  radios. 
The  TCMs  contained  the  carrier  sense  multiple-access  algorithm  used  by  the  TACFIRE  com¬ 
munications  protocol.  Each  node  also  contained  a  message  driver,  providing  communications 
loading,  and  data  collection  software  to  log  the  sending  and  receipt  of  messages,  as  well  as 
information  on  queues.  A  controlled  laboratory  experiment  was  conducted  to  quantify  the 
effects  of  message  length,  message  transmission  rate,  and  frequency  hopping  on  the  network 
throughput  and  delay  requirements  expected  to  be  placed  on  AFATDS  [5]. 

In  place  of  the  more  commonly  used  mission  threads,  a  scenario  generator  was  written  to 
create  “messages”  of  character  strings  (Os  and  Is)  of  a  specified  length  and  arrival  rate  ^  over 
a  1-hour  period.  The  intent  of  using  strings  is  to  emulate  the  anticipated  actual  operation 
of  a  network  without  incorporating  actual  “scripted”  scenarios.  Their  use  also  allows  the 
experimenter  to  easily  extend  the  range  of  consideration  to  include  communications  load 
levels  that  exceed  the  theoretical  capacity  of  the  network  in  question,  thus  more  expeditiously 
predicting  and  accurately  representing  dynamic  field  conditions. 

Four  message  arrival  rates  emulated  the  rate  of  actual  user-generated  messages  and  spe¬ 
cific  nodes’  ability  to  respond  to  incoming  messages.  Message  lengths  were  chosen  based  on 
the  message  lengths  used  by  Magnavox  (e.g.,  the  48-character  message  corresponded  to  the 

^The  theoretical  capacity  of  a  communications  network  is  commonly  expressed  in  terms  of  a  Poisson 
distributed  random  variable  with  exponentially  distributed  inter-arrival  times.  Even  though  we  know  that 
field  conditions  do  not  present  an  exponential  distribution  of  events,  it  is  the  best  with  which  we  have  to 
work. 
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Magnavox  576-bit  message).  The  messages  were  equally  distributed  among  the  four  nodes. 
For  example,  if  the  expected  arrival  rate  to  the  network  was  2,000  messages/hour,  the  sce¬ 
nario  generator  created  a  file  of  500  messages  for  each  node.  A  message  was  assumed  to 
enter  network  service  when  it  reached  the  modem. 

Once  the  message  was  generated,  the  communications  protocol  added  several  layers  of 
information  to  ensure  the  message  arrived  at  its  destination.  This  included  5  error  detec¬ 
tion/correction  bits  for  each  7-bit  character,  4  synchronization  characters,  and  a  preamble  to 
bring  the  transmitter  to  full  power  before  the  message  was  sent.  Acknowledgments  (ACKs), 
though  shorter  in  length,  were  wrapped  with  similar  overhead  bits.  Figure  1  illustrates  the 

Message  Length 


MESSAGE ; 


preamble 

sync  chars 

message  chars 

(0.6  sec) 

4  characters  (8  bits) 

48-352  characters  (12  bits) 

Acknowledgment  Length 


ACK: 

preamble 

sync  chars 

acknowledgement 

(0.6  sec) 

4  characters  (8  bits) 

16  characters  (12  bits) 

Figure  1. — The  Components  of  Messages  and  Acknowledgments 
message  and  acknowledgment  components. 

The  data  were  collected  in  accordance  with  an  incomplete  block  design  to  control  for  day- 
to-day  variability.  The  experiment  was  replicated  three  times  to  ensure  a  balanced  design 
[6]. 

The  results  of  the  AFATDS  experiment  with  actual  hardware  and  protocols  provided 
valuable  information  on  CNR  thresholds  and  showed  that  even  under  benign  laboratory 
conditions,  there  was  inherent  degradation,  discounting  the  modeling  assumption  of  “perfect’ 
communications.  The  analyses  concluded  that  SINCGARS  radios  employing  the  TACFIRE 
protocol  would  experience  difficulty  in  accommodating  the  expected  data  rates  of  AFATDS. 
Average  throughput  in  the  experiment  did  not  exceed  55%  of  the  network’s  capacity  (47% 
in  frequency  hopping  mode).  Utilization  never  exceeded  80%,  while  network  delays  were 
always  greater  than  twice  the  message  transmission  time. 

The  experiment  confirmed  some  information  that  was  considered  intuitive  (e.g.,  as  the 
message  arrival  rate  increased,  throughput  also  increased),  but  it  also  uncovered  several 
anomalies  of  the  TACFIRE  protocol  operating  with  SINCGARS  radios.  The  following  is  a 


6 


partial  list  of  some  of  those  anomalies;  1)  message  transmission  overhead  is  excessive;  in 
general,  the  time  to  transmit  the  preamble  is  longer  than  the  time  necessary  to  transmit 
the  actual  information,  2)  net  access  delay  overrides  message  priority  (i.e.,  the  node  with 
the  shortest  net  access  delay  basically  takes  control  of  the  entire  network  during  high  traffic 
periods;  prioritizing  messages  within  a  node  has  little  effect  unless  the  priority  scheme  is 
passed  to  the  modem  in  some  way),  3)  higher  bit  rates  do  not  linearly  increase  throughput, 
and  4)  frequency  hopping  increases  the  number  of  collisions. 

These  limits  that  were  identified  should  be  considered  when  modeling  or  designing  com¬ 
munications  architectures  and  protocols,  and  consideration  should  be  given  to  future  exper¬ 
imentation  to  evaluate  those  limits.  The  experimental  results  reaffirmed  those  critical  issues 
which  AFATDS  developers  addressed  during  the  development  of  this  complex  C3  system. 


Conclusion 


In  this  paper  we  have  shown  how  message  strings  may  be  used  to  simplify  the  testing  and 
evaluation  of  C2  systems  by  limiting  the  more  complex  mission  thread  testing  to  those  cases 
where  the  communications  equipment  can  support  the  message  load.  Testing  with  message 
strings  may  also  be  combined  with  theoretical  studies  and  simulations  as  shown  in  Figure  2 
to  provide  a  validated  basis  for  both  the  design  and  test  and  evaluation  of  C2  systems. 


Figure  2. — The  Complete  Simulation  Cycle 

Experimentation  provides  the  validated  parameters  for  the  theory  and  simulations,  which  in 
turn  provide  insights  to  the  system  performance  as  the  C2  system  is  scaled  to  operational 
sizes. 
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Abbreviations 


ACK 

Acknowledgment 

AF 

Adjust  Fire 

AFATDS 

Advanced  Field  Artillery  Tactical  Data  System 

ANOVA 

Analysis  of  Variance 

ARL 

Army  Research  Laboratory 

BRL 

Ballistic  Research  Laboratory 

btry 

battery 

C2 

Command  and  Control 

C3 

Command,  Control,  and  Communications 

CNR 

Combat  Net  Radio 

CPXRF 

Command  Post  Exercise  Research  Facility 

DMD 

Digital  Message  Device, 

EOM 

End  of  Mission 

FA 

Field  Artillery 

FDC 

Fire  Direction  Center 

FDO 

Fire  Direction  Officer 

FDT&E  II 

Force  Development,  Testing,  and  Experimentation  II 

FFE 

Fire  for  Effect 

FIST 

Fire  Support  Team 

FO 

Forward  Observer 

HEL 

Human  Engineering  Laboratory 

IDT 

Information  Distribution  Technology 

MOP 

Measure  of  Performance 

MTO 

Message  to  Observer 

PM 

Program  Manager 

SHOT 

Shot  out 

SINCGARS 

Single  Channel  Ground  and  Airborne  Radio  System 

TACFIRE 

Tactical  Fire  Direction  System 

TCM 

Tactical  Communications  Modem 
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